Devices and methods for presenting and regulating auxiliary information on an image display of a telesurgical system to assist an operator in performing a surgical procedure

ABSTRACT

Systems and methods for performing robotically-assisted surgical procedures on a patient enable an image display device to provide an operator with auxiliary information related to the surgical procedure, in addition to providing an image of the surgical site itself. The systems and methods allow an operator to selectively access and reference auxiliary information on the image display device during the performance of a surgical procedure.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.09/464,455, filed Dec. 14, 1999, now U.S. Pat. No. 6,522,906, which is acontinuation-in-part of U.S. patent application Ser. No. 09/457,406,entitled “Image Shifting Apparatus and Method for a Telerobotic System,”tiled Dec. 7, 1999, now U.S. Pat. No. 6,799,065, which claims priorityfrom U.S. provisional patent application Ser. No. 60/111,711, filed Dec.8, 1998, the full disclosures of which are incorporated herein byreference. This application is also related to the following patents andpatent applications, the full disclosures of which are incorporatedherein by reference: PCT International Application No. PCT/US98/19508,entitled “Robotic Apparatus,” filed on Sep. 18, 1998; U.S. patentapplication Ser. No. 09/418,726, entitled “Surgical Robotic Tools, DataArchitecture, and Use,” filed on Oct. 15, 1999; U.S. patent applicationSer. No. 09/378,173, entitled “Stereo Imaging System for Use inTelerobotic Systems,” filed on Aug. 20, 1999; U.S. patent applicationSer. No. 09/398,507, entitled “Master Having Redundant Degrees ofFreedom,” filed Sep. 17, 1999; U.S. patent application Ser. No.09/288,068, entitled “Aspects of a Control System of a MinimallyInvasive Surgical Apparatus;” riled Apr. 7, 1999; U.S. patentapplication Ser. No. 09/373,678, entitled “Camera Referenced Control ina Minimally Invasive Surgical Apparatus,” filed Aug. 13, 1999; U.S.patent application Ser. No. 09/398,960, entitled “Repositioning andOrientation of Master/Slave Relationship in Minimally InvasiveTelesurgey,” filed Sep. 17, 1999; and U.S. Pat No. 5,808,665, entitled“Endoscopic Surgical Instrument and Method for Use,” issued on Sep. 15,1998.

BACKGROUND OF THE INVENTION

The present invention is generally related to improved robotic devices,systems and methods, for use in telerobotic surgery.

Minimally invasive medical techniques are aimed at reducing the amountof extraneous tissue which may be damaged during diagnostic or surgicalprocedures, thereby reducing patient recovery time, discomfort, anddeleterious side effects. Many surgeries are performed each year in theUnited States. A significant amount of these surgeries can potentiallybe performed in a minimally invasive manner. However, only a relativelysmall percentage of surgeries currently use minimally invasivetechniques due to limitations of minimally invasive surgical instrumentsand techniques currently used and the difficulty experienced inperforming surgeries using such traditional instruments and techniques.

Advances in minimally invasive surgical technology could dramaticallyincrease the number of surgeries performed in a minimally invasivemanner. The average length of a hospital stay for a standard surgery issignificantly longer than the average length for the equivalent surgeryperformed in a minimally invasive surgical manner. Thus, expansion inthe use of minimally invasive techniques could save millions of hospitaldays, and consequently millions of dollars annually, in hospitalresidency costs alone. Patient recovery times, patient discomfort,surgical side effects, and time away from work can also be reduced byexpanding the use of minimally invasive surgery.

Traditional forms of minimally invasive surgery include endoscopy. Oneof the more common forms of endoscopy is laparoscopy, which is minimallyinvasive inspection or surgery within the abdominal cavity. Intraditional laparoscopic surgery a patient's abdominal cavity isinsufflated with gas and cannula sleeves are passed through small(approximately ½ inch) incisions in the musculature of the patient'sabdomen to provide entry ports through which laparoscopic surgicalinstruments can be passed in a sealed fashion.

The laparoscopic surgical instruments generally include a laparoscopefor viewing the surgical field and working tools defining end effectors.Typical surgical end effectors include clamps, graspers, scissors,staplers, and needle holders, for example. The working tools are similarto those used in conventional (open) surgery, except that the workingend or end effector of each tool is separated from its handle by anapproximately 12-inch long extension tube, for example, so as to permitthe surgeon to introduce the end effector to the surgical site and tocontrol movement of the end effector relative to the surgical site fromoutside a patient's body.

To perform surgical procedures, the surgeon typically passes theseworking tools or instruments through the cannula sleeves to the internalsurgical site and manipulates the instruments or tools from outside theabdomen by sliding them in and out through the cannula sleeves, rotatingthem in the cannula sleeves, levering (i.e., pivoting) the instrumentsagainst the abdominal wall and actuating the end effectors on the distalends of the instruments from outside the abdominal cavity. Theinstruments normally pivot around centers defined by the incisions whichextend through the muscles of the abdominal wall. The surgeon typicallymonitors the procedure by means of a television monitor which displaysan image of the surgical site via the laparoscopic camera. Typically,the laparoscopic camera is also introduced through the abdominal wall soas to capture an image of the surgical site. Similar endoscopictechniques are employed in, e.g., arthroscopy, retropentoneoscopy,pelviscopy, nephroscopy, cystoscopy, cistemoscopy, sinoscopy,hysteroscopy, urethroscopy, and the like.

There are many disadvantages relating to such traditional minimallyinvasive surgical (MIS) techniques. For example, existing MISinstruments deny the surgeon the flexibility of tool placement found inopen surgery. Difficulty is experienced in approaching the surgical sitewith the instruments through the small incisions. The length andconstruction of many endoscopic instruments reduces the surgeon'sability to feel forces exerted by tissues and organs on the end effectorof the associated instrument. Furthermore, coordination of the movementof the end effector of the instrument as viewed in the image on thetelevision monitor with actual end effector movement is particularlydifficult, since the movement as perceived in the image normally doesnot correspond intuitively with the actual end effector movement.Accordingly, lack of intuitive response to surgical instrument movementinput is often experienced. Such a lack of intuitiveness, dexterity andsensitivity of endoscopic tools has been found to be an impediment tothe expansion of the use of minimally invasive surgery.

Minimally invasive telesurgical systems for use in surgery have been andare still being developed to increase a surgeon's dexterity as well asto permit a surgeon to operate on a patient in an intuitive manner.Telesurgery is a general term for surgical systems where the surgeonuses some form of remote control, e.g., a servomechanism, or the like,to manipulate surgical instrument movements, rather than directlyholding and moving the tools by hand. In such a telesurgery system, thesurgeon is typically provided with an image of the surgical site on avisual display at a location remote from the patient. The surgeon cantypically perform the surgical procedure at the location remote from thepatient whilst viewing the end effector movement during the surgicalprocedure on the visual display. While viewing typically athree-dimensional image of the surgical site on the visual display, thesurgeon performs the surgical procedures on the patient by manipulatingmaster control devices at the remote location, which master controldevices control motion of the remotely controlled instruments.

Typically, such a telesurgery system can be provided with at least twomaster control devices (one for each of the surgeon's hands), which arenormally operatively associated with two robotic arms on each of which asurgical instrument is mounted. Operative communication between mastercontrol devices and associated robotic arm and instrument assemblies istypically achieved through a control system. The control systemtypically includes at least one processor which relays input commandsfrom the master control devices to the associated robotic arm andinstrument assemblies and from the arm and instrument assemblies to theassociated master control devices in the case of, e.g., force feedback,or the like.

One object of the present invention is to provide improved telesurgerysystems, devices and methods for use in surgery. Another object of theinvention is to provide a telesurgical system and method wherebyauxiliary information related to a surgical procedure to be performed bythe telesurgical system can be selectively displayed on a viewer of thesystem, together with an image of the surgical site captured by an imagecapture device, such as an endoscope, of the system, so as to enable anoperator of the system selectively to reference such auxiliaryinformation on the viewer during the performance of the surgicalprocedure. In this manner the surgical procedure can typically beperformed with greater confidence, safety, efficacy and in some casesgreater accuracy.

SUMMARY OF THE INVENTION

According to one aspect of the invention, there is provided a method ofperforming a surgical procedure on a patient. The method typicallycomprises positioning a surgical work station of a roboticallycontrolled surgical system and a patient on which a surgical procedureis to be performed in close proximity relative to each other, directinga field of view of an image capture device of the surgical work stationat a surgical site on the patient, at which site the surgical procedureis to be performed, and introducing at least one robotically controlledsurgical instrument on the surgical work station to the surgical site sothat an end effector of the surgical instrument is within the field ofview of the image capture device.

The method typically further comprises displaying an image of thesurgical site and the end effector on a display area of an image displayat an operator control station of the surgical system. The image displayis typically operatively connected to the image capture device so as todisplay, on the display area, the image of the surgical site and the endeffector captured by the image capture device. The method can furtherinclude permitting an operator of the surgical system to manipulate amaster control at the operator control station whilst viewing the imageof the surgical site on the image display, the master control beingoperatively associated with the surgical instrument thereby to cause theend effector to move in response to manipulation of the master controlso as to perform at least part of the surgical procedure on the patientat the surgical site.

The method yet further typically comprises operatively linking the imagedisplay to a source of selectively accessible auxiliary informationrelated to the surgical procedure to be performed, enabling the operatorselectively to access the source of auxiliary information from theoperator control station so as to forward the auxiliary information tothe image display, causing the auxiliary information to be displayedacross the display area of the image display in response to the operatorselectively accessing the source of auxiliary information at theoperator control station and enabling the operator selectively to movethe auxiliary information when displayed on the image display relativeto the image of the surgical site displayed on the image display.

The master control is typically operatively linked with the source ofauxiliary information, enabling the operator selectively to access thesource of auxiliary information then including permitting the operatorselectively to disassociate the master control from the surgicalinstrument and to use the master control to access the source ofauxiliary information so as to enable the auxiliary information to bedisplayed on the display area of the image display.

According to another aspect of the invention, there is provided a methodof performing a surgical procedure on a patient, the method comprisingmanipulating a master control whilst viewing a real time image of asurgical site on an image display, moving an end effector in response tomanipulation of the master control so as to perform at least part of asurgical procedure at the surgical site and selectively accessing asource of auxiliary information by means of the master control. Themethod typically further comprises displaying the auxiliary informationon the image display.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described, by way of example only, and withreference to the accompanying diagrammatic drawings, in which:

FIG. 1 shows a three-dimensional view of an operator control station, orsurgeon's console, and a surgical work station, or cart, of atelesurgical system in accordance with the invention, the cart carryingthree robotically controlled arms, the movement of the arms beingremotely controllable from the control station;

FIG. 2 shows, at an enlarged scale, a side view of a robotic arm andsurgical instrument assembly of the surgical station shown in FIG. 1;

FIG. 3 shows, at an enlarged scale, a three-dimensional view of atypical surgical instrument of the system shown in FIG. 1;

FIG. 4 shows a schematic kinematic diagram corresponding to the sideview of the robotic arm shown in FIG. 2, and indicates the arm havingbeen displaced from one position into another position;

FIG. 5 shows, at an enlarged scale, a wrist member and an end effectorof the surgical instrument shown in FIG. 3, the wrist member and the endeffector being movably mounted on a working end of a shaft of thesurgical instrument;

FIG. 6 shows a three-dimensional view of one of the master controldevices of the control station shown in FIG. 1, the master controldevice including a hand-held part, or wrist gimbal, and an articulatedarm portion on which the hand-held part is mounted;

FIG. 7 shows a schematic three-dimensional drawing indicating thepositions of the end effectors relative to a viewing end of an endoscopeon the surgical station and the corresponding positions of the mastercontrol devices relative to the eyes of an operator, typically asurgeon, at the control station;

FIG. 8 shows a schematic three-dimensional drawing indicating theposition and orientation of an end effector relative to a cameraCartesian coordinate reference system at a viewing end of the endoscope;

FIG. 9 shows a schematic three-dimensional drawing indicating theposition and orientation of a pincher formation of one of the mastercontrol devices relative to an eye Cartesian coordinate reference systemat a viewer of the control station;

FIG. 10 shows a block diagram indicating control steps of a controlsystem of the surgical system, the control system being arranged toeffect control between master control device input and correspondingsurgical instrument movement output;

FIG. 11 shows a flow diagram indicating control steps of a method, inaccordance with the invention, whereby an operator of the telesurgicalsystem can selectively access one or more sources of auxiliaryinformation related to a surgical procedure to be performed by thetelesurgical system, so as to display the information from the selectedsource or sources on an image display of the telesurgical systemtogether with an image of the surgical site captured by an imagecapturing device, such as an endoscope, of the system;

FIG. 12 shows a schematic view of an image of a surgical site displayedon the image display of the telesurgical system and further shows animage corresponding to auxiliary information from a selected source ofauxiliary information displayed in a window overlaid on the image of thesurgical site;

FIGS. 13A and B show schematic views illustrating the adjustment inposition and orientation of an image corresponding to auxiliaryinformation from a selected source of auxiliary information, relative toan image of the surgical site from an image capturing device;

FIG. 14 shows a schematic diagram of an image displayed at a viewer ofthe system shown in FIG. 1, and further shows a probe gatheringauxiliary information relating to a surgical procedure;

FIG. 15 shows a schematic side view of an image capturing device forcapturing an image along a shaft of a surgical instrument of the systemshown in FIG. 1, in accordance with the invention:

FIGS. 16A and 16B show schematic drawings indicating a rotationalmovement of a displayed image and a line or bar displayed in the imageto indicate orientational position of a scene in the displayed imagerelative to a world reference frame; and

FIGS. 17A and 17B show schematic drawings indicating a rotationalmovement of a displayed image and opposed markers or indicatorsdisplayed in the image to indicate orientational position of a scene inthe displayed image relative to a world reference frame.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIG. 1 of the drawings, a minimally invasive telesurgicalsystem, or robotically controlled surgical system, in accordance withthe invention is generally indicated by reference numeral 10. The system10 includes a control station, or surgeon's console, generally indicatedby reference numeral 12. The station 12 includes an image display orviewer 14 where an image of a surgical site is displayed in use. Asupport 16 is provided on which an operator, typically a surgeon, canrest his or her forearms while gripping two master control devices, oneof which is shown in FIG. 6 of the drawings, one in each hand. Themaster controls are positioned in a space 18 inwardly beyond the support16. When using the control station 12, the surgeon typically sits in achair in front of the control station 12, positions his or her eyes infront of the viewer 14 and grips the master controls one in each handwhile resting his or her forearms on the support 16.

The system 10 further includes a surgical work station, or cart,generally indicated by reference numeral 20. In use, the cart 20 ispositioned in close proximity to a patient requiring surgery and is thennormally caused to remain stationary until a surgical procedure to beperformed by means of the system 10 has been completed. The cart 20typically has wheels or castors to render it mobile. The station 12 istypically positioned remote from the cart 20 and can be separated fromthe cart 20 by a great distance, even miles away, but will typically beused within an operating room with the cart 20.

The cart 20 typically carries at least three robotic arm assemblies. Oneof the robotic arm assemblies, indicated by reference numeral 22, isarranged to hold an image capture device 24, e.g., an endoscope, or thelike. Each of the other two arm assemblies 26, 26 respectively, isarranged to hold a robotically controlled surgical instrument 28. Theendoscope 24 has an object viewing end 24.1 at a remote end of anelongate shaft thereof. It will be appreciated that the endoscope 24 hasan elongate shaft to permit its viewing end 24.1 to be inserted throughan entry port in a patient's body so as to access an internal surgicalsite. The endoscope 24 is operatively connected to the viewer 14 todisplay an image captured at its viewing end 24.1 on a display area ofthe viewer 14. Each robotic arm assembly 26, 26 is normally operativelyconnected to one of the master controls. Thus, the movement of therobotic arm assemblies 26, 26 is controlled by manipulation of themaster controls. The instruments 28 on the robotic arm assemblies 26, 26have end effectors which are mounted on wrist members which arepivotally mounted on distal ends of elongate shafts of the instruments28. It will be appreciated that the instruments 28 have elongate shaftsto permit the end effectors to be inserted through entry ports in apatient's body so as to access the internal surgical site. Movement ofthe end effectors relative to the ends of the shafts of the instruments28 is also controlled by the master controls. When a surgical procedureis to be performed, the cart 20 carrying the robotic arms 22, 26, 26 iswheeled to the patient and is normally maintained in a stationaryposition relative to, and in close proximity to, the patient, during thesurgical procedure.

In FIG. 2 of the drawings, one of the robotic arm assemblies 26 is shownin greater detail, and on an enlarged scale. Each assembly 26 typicallyhas a surgical instrument, schematically and generally indicated byreference numeral 28, releasably mounted thereon. FIG. 3 indicates thegeneral appearance of a typical surgical instrument 28 in greaterdetail.

Referring now to FIG. 3 of the drawings, the surgical instrument 28includes an elongate shaft 28.1. The wrist-like mechanism, generallyindicated by reference numeral 32, is located at a working end of theshaft 28.1. A housing 34, arranged releasably to couple the instrument28 to the robotic arm 26, is located at an opposed end of the shaft28.1. In FIG. 2, and when the instrument 28 is coupled or mounted on therobotic arm 26, the shaft 28.1 extends along an axis indicated at 28.2.The instrument 28 is typically releasably mounted on a carriage 36,which can be driven to translate along a linear guide formation 38 ofthe arm 26 in the direction of arrows P.

The robotic arm 26 includes a cradle, generally indicated at 40, anupper arm portion 42, a forearm portion 44 and the guide formation 38.The cradle 40 is pivotally mounted on a plate 46 in a gimbaled fashionto permit rocking movement of the cradle 40 about a pivot axis 48. Theupper arm portion 42 includes link members 42.1, 42.2 and the forearmportion 44 includes link members 44.1, 44.2. The link members 42.1, 42.2are pivotally mounted on the cradle 40 and are pivotally connected tothe link members 44.1, 44.2. The link members 44.1, 44.2 are pivotallyconnected to the guide formation 38. The pivotal connections between thelink members 42.1, 42.2, 44.1, 44.2, the cradle 40, and the guideformation 38 are arranged to constrain the robotic arm 26 to move in aspecific manner. The movement of the robotic arm 26 is illustratedschematically in FIG. 4.

With reference to FIG. 4 of the drawings, the solid lines schematicallyindicate one position of the robotic arm 26 and the dashed linesindicate another possible position into which the arm 26 can bedisplaced from the position indicated in solid lines.

It will be understood that the axis 28.2 along which the shaft 28.1 ofthe instrument 28 extends when mounted on the robotic arm 26 pivotsabout a pivot center or fulcrum 50. Thus, irrespective of the movementof the robotic arm 26, the pivot center 50 normally remains in the sameposition relative to the stationary cart 20 on which the arm 26 ismounted. In use, the pivot center 50 is positioned at a port of entryinto a patient's body when an internal surgical procedure is to beperformed. It will be appreciated that the shaft 28.1 extends throughsuch a port of entry, the wrist-like mechanism 32 then being positionedinside the patient's body. Thus, the general position of the mechanism32 relative to the surgical site in a patient's body can be changed bymovement of the arm 26. Since the pivot center 50 is coincident with theport of entry, such movement of the arm does not excessively effect thesurrounding tissue at the port of entry.

As can best be seen with reference to FIG. 4, the robotic arm 26provides three degrees of freedom of movement to the surgical instrument28 when mounted thereon. These degrees of freedom of movement arefirstly the gimbaled motion indicated by arrows A, pivoting or pitchingmovement as indicated by arrows B and the linear displacement in thedirection of arrows P. Movement of the arm 26 as indicated by arrows A,B and P is controlled by appropriately positioned actuators, e.g.,electrical motors, or the like, which respond to inputs from anassociated master control to drive the arm 26 to a required position asdictated by movement of the master control. Appropriately positionedsensors, e.g., potentiometers, encoders, or the like, are provided onthe arm 26 to enable a control system of the minimally invasivetelesurgical system 10 to determine joint positions as described ingreater detail herein below. It will be appreciated that whenever“sensors” are referred to in this specification, the term is to beinterpreted widely to include any appropriate sensors, such as, forexample, positional sensors, velocity sensors, or the like. It will beappreciated that by causing the robotic arm 26 selectively to displacefrom one position to another, the general position of the wrist-likemechanism 32 relative to the surgical site can be varied during theperformance of a surgical procedure

Referring now to FIG. 5 of the drawings, the wrist-like mechanism 32will now be described in greater detail. In FIG. 5, the working end ofthe shaft 28.1 is indicated at 28.3. The wrist-like mechanism 32includes a wrist member 52. One end portion of the wrist member 52 ispivotally mounted in a clevis, generally indicated at 54, on the end28.3 of the shaft 28.1 by means of a pivotal connection 56. The wristmember 52 can pivot in the direction of arrows D about the pivotalconnection 56. An end effector, generally indicated by reference numeral60, is pivotally mounted on an opposed end of the wrist member 52. Theend effector 60 is in the form of, e.g., a clip applier for anchoringclips during a surgical procedure, or the like. Accordingly, the endeffector 60 has two parts 60.1, 60.2 together defining a jaw-likearrangement.

It will be appreciated that the end effector 60 can be in the form ofany desired surgical tool, e.g., having two members, or fingers, whichpivot relative to each other, such as, for example, scissors, pliers foruse as needle drivers, or the like. Instead, it can include a singleworking member, e.g., a scalpel, cautery electrode, or the like. When atool other than a clip applier is required during the surgicalprocedure, the tool 28 is simply removed from its associated arm 26 andreplaced with an instrument bearing the required end effector, e.g., ascissors, or pliers, or the like.

The end effector 60 is pivotally mounted in a clevis, generallyindicated by reference numeral 62, on an opposed end of the wrist member52, by means of a pivotal connection 64. It will be appreciated thatfree ends 60.3, 60.4 of the parts 60.1, 60.2 are angularly displaceableabout the pivotal connection 64 toward and away from each other asindicated by arrows E, F. It will further be appreciated that themembers 60.1, 60.2 can be displaced angularly about the pivotalconnection 64 to change the orientation of the end effector 60 as awhole, relative to the wrist member 52. Thus, each part 60.1, 60.2 isangularly displaceable about the pivotal connection 64 independently ofthe other, so that the end effector 60, as a whole, is angularlydisplaceable about the pivotal connection 64 as indicated in dashedlines in FIG. 5. Furthermore, the shaft 28.1 is rotatably mounted on thehousing 34 for rotation as indicated by the arrows G. Thus, the endeffector 60 has three degrees of freedom of movement relative to the arm26, namely, rotation about the axis 28.2 as indicated by arrows G,angular displacement as a whole about the pivot 64 and angulardisplacement about the pivot 56 as indicated by arrows D. By moving theend effector 60 within its three degrees of freedom of movement, itsorientation relative to the end 28.3 of the shaft 28.1 can selectivelybe varied. It will be appreciated that movement of the end effector 60relative to the end 28.3 of the shaft 28.1 is controlled byappropriately positioned actuators, e.g., electrical motors, or thelike, which respond to inputs from the associated master control todrive the end effector 60 to a required orientation as dictated bymovement of the master control. Furthermore, appropriately positionedsensors, e.g., encoders, or potentiometers, or the like, are provided topermit the control system of the minimally invasive telesurgical system10 to determine joint positions as described in greater detail hereinbelow.

One of the master controls is indicated generally in FIG. 6 by referencenumeral 70. A hand held part, or wrist gimbal, of the master control 70is generally indicated by reference numeral 80. Part 80 has anarticulated arm portion including a plurality of members or links 82connected together by pivotal connections or joints 84. The surgeongrips the part 80 by positioning his or her thumb and index finger overa pincher formation 86. When the pincher formation 86 is squeezedbetween the thumb and index finger, the fingers or end effector elements60.1, 60.2 of the end effector 60 close. When the thumb and index fingerare moved apart, the fingers 60.1, 60.2 of the end effector 60 moveapart in sympathy with the moving apart of the pincher formation 86. Thejoints 84 of the part 80 are operatively connected to actuators, e.g.,electric motors, or the like, to provide for, e.g., force feedback,gravity compensation, and/or the like. Furthermore, appropriatelypositioned sensors, e.g., encoders, or potentiometers, or the like, arepositioned on each joint 84 of the part 80, so as to enable jointpositions of the part 80 to be determined by a control system of thesurgical system 10.

The part 80 is typically mounted on an articulated arm 90. Thearticulated arm 90 includes a plurality of links 92 connected togetherat pivotal connections or joints 94. It will be appreciated that alsothe articulated arm 90 has appropriately positioned actuators, e.g.,electric motors, or the like, to provide for, e.g., force feedback,gravity compensation, and/or the like. Furthermore, appropriatelypositioned sensors, e.g., encoders, or potentiometers, or the like, arepositioned on the joints 94 so as to enable joint positions of thearticulated arm 90 to be determined by the control system as describedin greater detail herein below.

To move the orientation of the end effector 60 and/or its position alonga translational path, the surgeon simply moves the pincher formation 86to cause the end effector 60 to move to where he wants the end effector60 to be with reference to the image viewed at the viewer 14. The endeffector position and/or orientation can be arranged to follow that ofthe pincher formation 86.

The actuators and sensors associated with the robotic arms 26, 26 andthe surgical instruments 28, 28 mounted thereon, and the actuators andsensors associated with the master control devices 70, 70 areoperatively linked in the control system. The control system typicallyincludes at least one processor, typically a plurality of processors,for effecting control between master control device input and responsiverobotic arm and surgical instrument output and for effecting controlbetween robotic arm and surgical instrument input and responsive mastercontrol output in the case of, e.g., force feedback, or the like.

In use, and as schematically indicated in FIG. 7 of the drawings, thesurgeon views the surgical site through the viewer 14. The end effector60 carried on each arm 26 is caused to perform positional andorientational movements in response to movement and action inputs on itsassociated master control. The master controls are indicatedschematically at 70, 70. It will be appreciated that during a surgicalprocedure images of the end effectors 60 are captured by the endoscope24 together with the surgical site and are displayed on the viewer 14 sothat the surgeon sees the responsive movements and actions of the endeffectors 60 as he or she controls such movements and actions by meansof the master control devices 70, 70. The control system is typicallyarranged automatically to cause end effector orientational andpositional movement as viewed in the image at the viewer 14 to be mappedonto orientational and positional movement of the pincher formation 86of its associated master control 70, as will be described in greaterdetail herein below.

The operation of the control system of the surgical system or apparatus10 will now be described. In the description which follows, the controlsystem will be described with reference to a single master control 70and its associated robotic arm 26 and surgical instrument 28. The mastercontrol 70 will be referred to simply as “master” and its associatedrobotic arm 26 and surgical instrument 28 will be referred to simply as“slave.”

The method whereby control between master movement and correspondingslave movement is achieved by the control system of the minimallyinvasive surgical apparatus will now be described with reference toFIGS. 8 to 10 of the drawings in overview fashion. For a more detaileddescription of control between master movement and corresponding slavemovement refer to Applicant's co-pending U.S. application Ser. No.09/373,678, entitled “Camera Referenced Control in a Minimally InvasiveSurgical Apparatus,” filed Aug. 13, 1999, which is fully incorporatedherein by reference as if part of this specification.

Control between master and slave movement is achieved by comparingmaster position and orientation in an eye Cartesian coordinate referencesystem with slave position and orientation in a camera Cartesiancoordinate reference system. For ease of understanding and economy ofwords, the term “Cartesian coordinate reference system” will simply bereferred to as “frame” in the rest of this specification. Accordingly,when the master is stationary, the slave position and orientation in thecamera frame is compared with the master position and orientation in theeye frame, and should the position and/or orientation of the slave inthe camera frame not correspond with the position and/or orientation ofthe master in the eye frame, the slave is caused to move to a positionand/or orientation in the camera frame at which its position and/ororientation in the camera frame does correspond with the position and/ororientation of the master in the eye frame. In FIG. 8, the camera frameis generally indicated by reference numeral 110 and the eye frame isgenerally indicated by reference numeral 150 in FIG. 9.

When the master, or pincher formation 86, is moved into a new positionand/or orientation in the eye frame 150, the new master position and/ororientation does not correspond with the previously corresponding slaveposition and/or orientation in the camera frame 110. The control systemthen causes the slave to move into a new position and/or orientation inthe camera frame 110 at which new position and/or orientation, itsposition and orientation in the camera frame 110 does correspond withthe new position and/or orientation of the master in the eye frame 150.

It will be appreciated that the control system includes at least one,and typically a plurality of processors which compute new correspondingpositions and orientations of the slave in response to master movementinput commands on a continual basis at a rate corresponding to theprocessing cycle rate of the control system. A typical processing cyclerate of the control system is about 1300 Hz. Thus, when the master ismoved from one position to a next position, the corresponding movementof the slave to respond is computed at about 1300 Hz. Naturally, thecontrol system can have any appropriate processing cycle rate dependingon the processor or processors used in the control system.

The camera frame 110 is typically positioned such that its origin 112 isat the viewing end 24.1 of the endoscope 24. Conveniently, the z axis ofthe camera frame 110 extends axially along a viewing axis 114 of theendoscope 24. Although, in FIG. 8, the viewing axis 114 is shown incoaxial alignment with a shaft axis 115 of the endoscope 24, it is to beappreciated that the viewing axis 114 can be angled relative thereto.Thus, the endoscope can be in the form of an angled scope. Naturally,the x and y axes are positioned in a plane perpendicular to the z axis.The endoscope is typically angularly displaceable about its shaft axis.The x, y and z axes are fixed relative to the viewing axis 114 of theendoscope 24 so as to displace angularly about the shaft axis insympathy with angular displacement of the endoscope 24 about its shaftaxis 115.

To enable the control system to determine slave position andorientation, a frame is defined on, or attached to, the end effector 60.This frame is referred to as an end effector frame or slave tip frame,in the rest of this specification and is generally indicated byreference numeral 116. Conveniently, the end effector frame 116 has itsorigin at the pivotal connection 64. However depending on the type ofend effector used, the origin may be offset relative to such a pivotalconnection should an improved or more intuitive response between masterinput and slave output be achieved thereby. For the end effector 60 asshown in the drawings, one of the axes, e.g., the z axis, of the frame116 is defined to extend along an axis of symmetry, or the like, of theend effector 60. Naturally, the x and y axes then extend perpendicularlyto the z axis. It will be appreciated that the orientation of the slaveis then defined by the orientation of the frame 116 having its origin atthe pivotal connection 64, relative to the camera frame 110. Similarly,the position of the slave is then defined by the position of the origin118 of the frame 116 relative to the camera frame 110.

Referring now to FIG. 9 of the drawings, the eye frame 150 is typicallychosen such that its origin corresponds with a position 152 where thesurgeon's eyes are normally located when he or she is viewing thesurgical site at the viewer 14. The z axis typically extends along aline of sight of the surgeon, indicated by axis 154, when viewing thesurgical site through the viewer 14. Naturally, the x and y axes extendperpendicularly from the z axis at the origin 152. Conveniently, the yaxis is chosen to extend generally vertically relative to the viewer 14and the x axis is chosen to extend generally horizontally relative tothe viewer 14.

To enable the control system to determine master position andorientation in the viewer frame 150, an appropriate point, e.g., point3A, is chosen on the master to define an origin 156 of a master ormaster tip frame 158. It will be appreciated that the point relative tothe master at which the origin 156 of the master frame 158 is attachedis chosen to enhance intuitive response between master and slave and canthus be at any appropriate location relative to the master.Conveniently, the z axis of the master frame 158 on the master extendsalong an axis of symmetry of the pincher formation 86 which extendscoaxially along a rotational axis H of the pincher formation 86 relativeto the rest of the master 70. The x and y axes then extendperpendicularly from the rotational axis H at the origin 3A.Accordingly, orientation of the master within the eye frame 150 isdefined by the orientation of the master frame 158 relative to the eyeframe 150. The position of the master in the eye frame 150 is defined bythe position of the origin 156 at 3A relative to the eye frame 150.

Referring now to FIG. 10 of the drawings, a control system employed tocause the slave to track master input is generally and schematicallyindicated by reference numeral 200. The control method as indicated byreference numeral 200 assumes that the master and slave were atcorresponding positions and the master has been moved into a newposition and orientation. Accordingly, since the new position andorientation of the pincher formation 86 relative to the eye frame 150 nolonger corresponds with the position and orientation of the end effectorframe 116 relative to the camera frame 110, the end effector 60 iscaused to move into a corresponding new position and orientationrelative to the camera frame 110 at which it does correspond with thenew position and orientation of the pincher formation 86 relative to theeye frame 150.

The new position and orientation of the pincher formation 86 is read injoint space as indicated by reference numeral 202. This is achieved bythe processor by means of the sensors operatively associated with thejoints on the master. From this joint space information, whichdetermines the joint positions of the master, a corresponding newposition and orientation of the master frame 158 relative to the eyeframe 150 is determined in Cartesian space as indicated by referencenumeral 204. In similar fashion, the current position and orientation ofthe end effector 60 in joint space is read as indicated by referencenumeral 206. From this information the current position and orientationof the end effector frame 116 relative to the camera frame 110 inCartesian space is computed, as indicated by reference numeral 208. Thenew position and orientation of the master frame 158 relative to the eyeframe 150 in Cartesian space is then compared with the current positionand orientation of the end effector frame 116 relative to the cameraframe 110 as indicated at 210. An error between the end effector frame116 current position and orientation relative to the camera frame 110and the position and orientation of the end effector frame 116 relativeto the camera frame 110 at which it would correspond with the newposition and orientation of the master frame 158 relative to the eyeframe 150 is then computed, as indicated at 212.

It will be appreciated that master orientational and positional movementvariation need not necessarily correspond proportionally with responsiveend effector orientational and positional movement variation.Accordingly, the system is typically arranged to provide for scaling sothat the translational movement of the end effector in response totranslational movement input on the master is scaled e.g., at a ratio 1to 2, or the like.

From the error, corresponding end effector command signals are computedas indicated at 214. The end effector command signals are then forwardedto the slave actuators to cause them to move the end effector 60 to anew position and orientation relative to the camera frame 110 at whichit corresponds with the new master position and orientation relative tothe eye frame 150, as indicated at 216. For further detail, refer to theSer. No. 09/373,678 application mentioned above.

A method and system whereby auxiliary information related to a surgicalprocedure to be performed by the system 10 can be selectively displayedon the viewer 14, together with an image of the surgical site capturedby the endoscope 24, so as to enable the surgeon selectively toreference such information on the viewer 14 during the performance ofthe surgical procedure, in accordance with the invention, will now bedescribed.

By displaying auxiliary information related to the surgical procedure inthe image of the surgical site displayed at the viewer 14, the surgeonis able to reference such information without having to look at anothersource or display. For example, by displaying a patient's ECG signal inthe image together with the image of the surgical site captured by theendoscope 24, the surgeon need not transfer his direction of view to alocation removed from the image of the surgical site. This enables thesurgeon to perform the surgical procedure with greater ease andconfidence and with less distraction. Furthermore, the surgeon canprepare preoperative information specific to the surgical procedure tobe performed, or specific to the patient on which the surgical procedureis to be performed, so as to enable the surgeon selectively to accesssuch specific auxiliary information in the displayed image during theperformance of the actual surgical procedure. When displaying theauxiliary information together with the image of the surgical sitecaptured by the endoscope is referred to in this specification, such adescription is to be interpreted to have a wide meaning including, forexample, displaying the image in a discrete window overlaid on the imageof the surgical site, displaying the auxiliary information so as to bemerged with the image of the surgical site, such as merging apreoperative x-ray image with the image of the surgical site so that thesurgeon can view hidden detail of the surgical site, displaying theauxiliary information selectively on the viewer instead of the image ofthe surgical site so that the surgeon is presented with an unobstructedview of the surgical site when performing the surgical procedure, theauxiliary information then being selectively displayable in the image atthe viewer alternately with the image of the surgical site, and thelike. It will be appreciated that the auxiliary information can bedisplayed on a separate image display or viewer where appropriate.

Referring to FIG. 11 of the drawings, a plurality of sources oftwo-dimensional information is generally indicated by reference numeral312. Another plurality of sources of two-dimensional information isgenerally indicated by reference numeral 314.

The sources of two dimensional auxiliary information at 312 defineauxiliary information to be displayed in the image at the viewer 14 andwhich is of a type which, when displayed in the image, is to beadjustable to vary its displayed position relative to the image of thesurgical site captured by the endoscope. The imaged information from 312is typically adjustable relative to the image of the surgical site intwo dimensions only. Accordingly, the position of the imaged informationcan be varied to change its position across the image of the surgicalsite.

If the imaged information from 312 is displayed in a window overlaid onthe image of the surgical site, the size of the window is typically alsoadjustable in two dimensions. The types of information selectivelyaccessible from the sources 312 include, for example, a prerecordedstreaming video of the surgical procedure to be performed so that theoperator can follow the procedure as depicted in the video whiledisplayed in the image at the viewer 14 together with the image of thesurgical site. The types of information can further include, forexample, a real time ECG signal so that the surgeon can monitor thepatient's heart beat within the displayed image at the viewer 14.

Another type of auxiliary information can be in the form of a previouslycaptured and stored image from the endoscope of the surgical site,wherein the pre-captured image was taken to provide a generallypanoramic view of the surgical site and the surrounding scene. Such apre-captured panoramic image can be obtained by the endoscope 24. Insuch a case, the image can be captured when the viewing end of theendoscope 24 is relatively far removed from the surgical site. After thepanoramic image or view is captured in this fashion, the endoscope canbe moved such that its viewing end is closer to the surgical site so asto obtain a more suitable real time image for use in the performance ofthe actual surgical procedure.

It will be appreciated that images other than a panoramic image of thesurgical site and surrounding scene can be provided for selectivereference on the image display at the viewer 14. Such other images caninclude, for example, generic or patient specific anatomical images foraiding the operator, or surgeon, for example, in identifying structuresso as to determine the surgical site location relative to the patientanatomy. Furthermore, such images can include, for example, imagesshowing the location of the entry ports, or incision points, theposition of the surgical instrument shafts and/or the end effectors soas to provide the operator with visible information relating to thelocation of surgical instruments or parts thereof. Such image can becomputer generated where appropriate, or can be obtained from additionalimage capture devices, and/or the like. This can be useful to avoidcollisions between the instrument shafts, for example. Furthermore, thiscan provide the operator with visible information enabling him toperceive how the instruments are interacting with each other and/or thepatient, in addition to the real time image of the surgical site used toperform the actual surgical procedure. When this information isselected, the auxiliary information can be displayed, where appropriate,to surround or abut a generally closer view of the surgical sitecaptured continually, or in real time, by the endoscope and which isused by the surgeon to monitor and control the surgical procedure. Inthis manner the surgeon, or operator, can be provided with the real timeimage from the endoscope at a preferably generally centrally disposedlocation in the viewed image, while the pre-captured, or real time,auxiliary image, e.g., a more panoramic view of the surgical site andsurrounding scene, is displayed along the periphery of the real timeimage obtained from the endoscope 24. This can serve to provide theoperator with a better idea of where he or she is operating relative tothe area surrounding the surgical site. Instead of providing theauxiliary image to surround the real-time image of the surgical site,the auxiliary image can be displayed in a discrete window, or in a“picture in picture” arrangement, extending over the image of thereal-time surgical site image. As another alternative, the auxiliaryimage can be displayed alternately with the actual real-time image.Thus, during the performance of a surgical procedure the surgeon canintermittently switch between the image of the real-time surgical siteimage and the auxiliary image by means of any appropriate switchinginput device or method, such as, buttons, switches, voice command,and/or the like. When the information from 312 is displayed in a windowoverlaid on the image of the surgical site, the surgeon can typicallyvary the size of the window and place the window relative to the imageof the surgical site so that the information is presented at a locationwhich is comfortable to the surgeon and at which the window does notobstruct important detail of the surgical site image.

By way of example, a specific application of such a “picture in picture”arrangement will now be described. During the course of a surgicalprocedure, the displayed image of the surgical site is typically in theform of a “narrow” field of view image normally being live, e.g.,continually updated, magnified and focused particularly on the surgicalsite. Such a “narrow” field of view typically provides the operator witha large image of a relatively small area in the patient. Such a “narrow”field image is typically captured in real time by means of the endoscope24. It has been found advantageous to provide the operator with a “wideangle” image of the surgical site and surrounding scene, to assist theoperator in determining where the surgical site and surgical tools arewith reference to the surrounding scene. Such a “wide angle” image canbe in the form of a “still” image captured by the same endoscope at aposition further removed from the surgical site than at which it isnormally positioned when capturing the real time image used by theoperator as he or she performs the surgical procedure. Instead, the“wide angle” image can be captured in real time by another image capturedevice, or endoscope, or the like. The two images can be displayed in avariety of different ways. In one way, the “wide angle” image can bedisplayed in a “smaller” window and the “narrow” field image can bedisplayed over a relatively larger area. The surgeon can then refer tothe “smaller” window for referencing orientation, or the like. Inanother way, the “narrow” field image is displayed in a “smaller” windowand the “wide angle” image is displayed over a relatively “larger” areato provide context to the surgeon to help him or her to remain orientedat the surgical site.

It can happen that the surgeon wishes to change the image displayed onthe viewer 14. This can be achieved, e.g., by rotation of the endoscope24 relative to the site viewed. Where the “wide angle” image is a“still” image, this image can be caused to rotate together with rotationof the “live”, magnified image. This can be achieved by causing the“still” image to be modified, for example, by means of computer control,so that the “still” image rotates to the same degree as the “live”image, so as to maintain, for example, context for the surgeon shouldthe surgeon desire to rotate the endoscope during surgery. In addition,or instead, if the surgeon desires to pan with the endoscope, the“still” image can be modified so that the “still” image preservesalignment, or registration, with a corresponding part of the “live”image.

The sources of two dimensional auxiliary information at 314 defineauxiliary information to be displayed in the image at the viewer 14 andwhich is of a type which, when displayed in the image, is to beadjustable to vary not only its two-dimensional displayed positionrelative to the image of the surgical site captured by the endoscope,but also its displayed orientation in three dimensions relative to thedisplayed image of the surgical site. One of the sources at 314 cancontain preoperative information which is to be aligned or brought intoregister with the image of the surgical site. For example, a twodimensional CAT scan image of a surgical site particular to the patienton which the surgical procedure is to be performed can be obtainedpreoperatively and loaded into one of the sources at 314. Such apreoperative image can be obtained so as to correspond with an image tobe captured by the endoscope, in other words, an image corresponding tothe image which the endoscope is to capture during the surgicalprocedure from a specific vantage point. Instead, the preoperative imagecan be from a vantage point different to that of where the endoscope isto be during the surgical procedure. During the surgical procedure, thesurgeon can then access the CAT scan information from the particularsource at 314 and place it in the displayed image of the surgical site.Such an image can then be adjusted in three dimensions so as to bringthe preoperative CAT scan image generally into register with the imageof the actual surgical site captured by the endoscope. Since theinformation from the sources 314 represent two dimensional information,there may be a limit to the amount of orientation change that can betolerated before the information ceases to be of use to the surgeon.

Still referring to FIG. 11 of the drawings, a plurality of sources ofthree-dimensional information is indicated at 316. One of the sourcescan include, for example, a three-dimensional model corresponding to asurgical site on which a surgical procedure is to be performed. Such athree-dimensional model can be, for example, raw volumetric images, suchas point cloud or voxcel representations, or the like, a computergenerated three-dimensional model or image, a segmentedthree-dimensional model obtained from CAT (Computer Aided Tomography)scans, MRI (Magnetic Resonance Imaging) techniques, or the like. Duringthe surgical procedure, the surgeon can then access the model and placeit in the image of the surgical site. The image corresponding to theauxiliary information in the form of the three-dimensional model, cantypically be superimposed, or merged, with the image of the surgicalsite. The brightness of the image of the three-dimensional model istypically adjustable so as to cause it selectively to fade relative tothe actual image of the surgical site.

Once placed in the image, the image of the model can be positionally andorientationally adjusted, and typically scaled, so as to enable thesurgeon to bring the preoperative image into register with the actualimage of the surgical site. Should the position of the endoscope bechanged, for example, to obtain an image of the surgical site from adifferent vantage point, the registration of the preoperative image canbe made to remain in register with the surgical site. This can typicallybe accomplished by causing the control system of the surgical system 10to fix the position of the preoperative image relative to a suitablereference frame once the surgeon has brought the preoperative imagegenerally into register in the displayed image. A suitable referenceframe can be, for example, a reference frame attached relative to thecart 20, or the patient, or the like. Since registration is ofteneffected visually by the surgeon, it may be that the registration is notentirely true or accurate. Thus, should the endoscope position be movedto capture an image of the surgical site from a different vantage point,it may be that the surgeon may again have to perform a slight adjustmentto the registration should the preoperative image not be correctlyregistered with the actual image of the surgical site upon changing theendoscope position. Instead of manual registration as described above,automatic registration of the preoperative image with the surgical siteimage can be achieved in accordance with known imaging techniques.Advantageously, registration can be accomplished by enabling thesurgeon, or operator, to perform an initial manual registrationprocedure, followed by an automatic registration procedure in accordancewith conventional methods to achieve a truer registration. Althoughreference has been made to a model, it will be appreciated that otherauxiliary information can be used instead. Such other auxiliaryinformation can include preoperative images as well as inter-operativeimages. For example, an inter-operative image, or preoperativelyobtained model, and/or the like, of a beating heart can be registeredwith the actual image of the beating heart as captured by the endoscope,and/or the like.

Referring again to the two-dimensional information at the sources 312,the two dimensional information can typically be in the form ofintrinsically two-dimensional information. Such information can includetwo dimensional visual images, such as video images, x-ray images,ultrasonic images, and/or the like. These two-dimensional images can bein digital or analog format, or the like. The information can be in theform of static images. Such static images can be in tiff, jpeg, and/orthe like, file formats, for example. The information can be in the formof moving images, such as, for example, streaming videos, as alreadymentioned. Such moving images can be in mpeg, digital video, analogvideo, such as NTSC or PAL, and/or the like, formats, for example. Theinformation can be textual, numeric, symbolic, and/or graphic in form.For example, the information sources can include sources of informationin the form of words, numeric readouts, status icons, bargraphs,stripchart displays, and/or the like. In this manner, for example,representations of blood pressure gauges, heartbeat rate, warmingmessages, notifications, warming lights, warning icons, or other warmingsignals related to system status, for example, the time in the form of arepresentation of a digital or analog clock, e-mail messages, and/or thelike, can be displayed. Accordingly, numeric readouts can correspond toblood pressure, heartbeat rate, elapsed and absolute time, and/or thelike. Status icons can include icons indicating the status of the system10, the identification of the type of surgical instruments currentlymounted on the robotic arms, and/or the like. Bar graphs can correspondto patient specific information, such as, temperature, oxygen levels inthe patient's blood, and/or the like. Bar graphs can also correspond tosystem specific information such as force magnitude, back-up batterystatus, and/or the like. Strip charts can correspond to EEG, ECG, bloodpressure, and/or the like. Symbolic or graphic representations cancorrespond to clocks, warning indicators, and icons selectivelyactivatable to provide access to sources of other auxiliary information,such as the three-dimensional and two-dimensional information, describedabove, menus, web pages and/or the like.

One, or more, of the sources may even comprise a separate computeroperatively connected to the system 10. The computer can be a computeron which a surgeon has prepared preoperative information for a specificpatient on which a surgical procedure using the system 10 is to beperformed. Such a computer may be remote from the system 10. When linkedto the system 10 as a source of auxiliary information, in accordancewith the invention, the surgeon is able to access such preoperativeinformation on the remote computer from the system 10, so as selectivelyto display such information on the viewer 14 during the performance ofthe surgical procedure. Thus, the surgeon, from this source, can accessinformation which may be resident on a computer screen within his or heroffice, for example.

The images derived from the sources at 312, 314 and/or 316, may bestored images or may be real-time images. Accordingly, the system 10 mayinclude dedicated memory on which the images can be recordedpreoperatively if the images are patient or surgical site specific, forexample, so as to be stored in resident memory of the system 10.Instead, or in addition, the system 10 can have one or more inputconnectors, or jacks, to enable the system 10 to be operatively linkedto a source of auxiliary information external to the system 10. In thisfashion, the system can be linked to an external source of auxiliaryinformation, such as, for example, a remote computer as described above,an ECG source, computer networks such as Local Area Networks (LANS), theinternet, and/or the like. Accordingly, it will be appreciated that thesources 312, 314 and 316, can be in the form of resident memory of thesystem 10, on which memory the auxiliary information is stored, or canbe in the form sources external to the system 10, which external sourcesare connectable to the system 10 through the input connectors or jacks.

Sources of three-dimensional information are indicated at 316. Thesesources represent information which is intrinsically three-dimensional.Such types of information can include, for example, segmented organand/or vasculature models, patient specific and/or generic biomedicalmodels, non-biological geometric shapes, markers, and/or the like. Suchtypes of information can also include, for example, real timethree-dimensional video, laser scans, and/or the like. Such types ofinformation can yet further include landmarks, identifiers, or othermarkers that are attached to fixed locations in space. The use of suchlandmarks, identifiers, or other markers will now be described, by wayof example. In the case where the surgeon wishes to perform ananastomosis, for example, he or she can place a landmark, or identifier,or the like in the image displayed on the image display and then movethe landmark or marker to correspond with the area where the anastomosisis to be performed. The marker can then be attached to the area so thatif the endoscope is moved, for example, the marker remains in aregistered condition with the area to which it is attached.

The non-biological geometric shapes are typically used to place visiblehaptic constraints in the displayed image at the viewer 14. The purposeof placing such haptic constraints in the image is, for example, toinhibit the end effectors from moving beyond such constraints,containing end effector movement within such constraints, and/or thelike. Accordingly, the operator of the system can select anappropriately shaped geometric shape, or shapes, and, place it, or them,in the image, and then position the selected geometric shape, or shapes,in the image around an area, or organ, or tissue, for example, so as toprotect that area, or organ, or tissue from invasion by the endeffectors 60, or to constrain end effector movement to remain withinsuch shape or shapes, miter-box-fashion. Thus, should the site on whichit is desired to perform a surgical procedure be close to a sensitiveorgan, or tissue, or the like, an appropriately shaped geometric shape,or shapes, can be selected, placed in the scene of the surgical site andmoved into a position in which the selected shape, or shapes, extendover the sensitive area. When the shape, or shapes, is so placed, acorresponding haptic constraint, corresponding to the selected andplaced geometric shape, or shapes, is initialized so as to inhibit theend effectors 60 from trespassing beyond the visible constraint, orconstraints, as placed in the image by the surgeon thereby to protectthe sensitive tissue, or organ, or the like. The geometric shapes can beof any appropriate shape. Accordingly, such shapes can include, forexample, polyhedral shapes, NURBS (Non-Uniform Rational B-Spline),implicit surface shapes, planar shapes such as walls, and/or the like.The geometric shapes can include volumetric shapes such as point cloud,voxcels, and/or the like. The file formats used to store such geometricshapes can be .obj, .dxf, .3ds, VRML, and/or the like, for example. Itwill be appreciated that once an appropriate selected geometric shape,or shapes, is placed in the image, the surgeon can move the shape, orshapes, into a position covering or shrouding an area of sensitivity.When this has been done, the control system of the system 10 cantypically allocate coordinates to the placed shape, or shapes, relativeto an appropriate frame, such as a frame attached to the cart 20, orpatient, or the like. The system, after having determined thecoordinates corresponding to the placed shape, or shapes, then inhibitsthe end effectors from moving beyond such coordinates or constrains endeffector movement to remain within such coordinates. For a more detaileddescription of a control system of the system 10 whereby suchconstraints can be imposed, refer to Applicant's co-pending applicationSer. No. 09/288,068 filed Apr. 7, 1999 entitled “Aspects of a ControlSystem of a Minimally Invasive Surgical Apparatus”. Geometric shapes canalso be used to guide the surgeon or to assist in finding locations ofparticular interest. Furthermore, haptic feedback can be used toindicate information about objects which may not be readily discernablevisually. For example, sensitive areas can be given repulsive behaviorso that the tools are not only inhibited from approaching the sensitiveareas, but are restrained when approaching the sensitive areas at apredetermined distance from such areas.

Such geometric shapes can be provided with geometric description oradditional information, and can contain information about appearance,e.g., via visual texture mapping, and/or the like, surface and volumeproperties, e.g., such as mass, density, impedance, and/or the like, inaccordance with known methods in the field of haptics. The shapes canalso be derived from biological sources such as segmented MRIs. Suchadditional information about geometric shapes can be used for visualrepresentation, e.g., colors, patterns, textual maps, flashingappearances, and/or the like. Such additional information can also beused with haptic rendering to provide, for example, stiffness,artificial friction, masses, vibrations, or other physical ornon-physical force cues.

The various sources of information as indicated at 312, 314, and 316,are typically represented as icons on the display area of the videodisplay 14. Accordingly, the operator of the system can select any oneor more of the desired sources by selecting the appropriate associatedicon. The step of selecting the desired source of auxiliary informationis indicated by the blocks 318, 320, and 322 for the sources at 312,314, and 316, respectively. Selection of a desired source typicallytakes place at the operator console 12. Such selection can be made inany appropriate manner, such as by using buttons, foot pedals, a mouse,and/or the like, for example. Advantageously, such selection is made bymaking use of one, or both, or either of the master controls 70, 70. Insuch a case, one, or both, or either, of the masters 70, 70 can serve asa two-dimensional or three-dimensional mouse. Accordingly, one, or both,or either, of the masters can be arranged to perform functions relativeto the displayed image in a manner analogous to a conventional mouserelative to a computer screen. Therefore, one, or both, or either, ofthe masters can be arranged to perform functions such as to point,highlight, move, select, and/or the like.

The masters each typically have at least six degrees of freedom ofmovement. Accordingly, when used as a three-dimensional mouse, suchmaster can be arranged to control six variables, for example. Therefore,functions such as, shifting, rotating, panning, tilting scaling, and/orthe like, can be performed simultaneously when one, or both, or either,of the masters are used as a three-dimensional mouse, without anotherinput being required. In particular, for two-handed or two-masteroperation, any windows or overlays can be handled as “elastic” bodies,such that resizing, scaling, warping, and/or the like, can, for example,be controlled by pulling the masters apart, or the like. In this manner,the selected auxiliary information when displayed in the display imageof the viewer 14 can be positionally and orientationally adjusted inthree-dimensions in a three-dimensional environment, where appropriate,or where desired. The masters 70, 70 are typically provided with forcefeedback. The force feedback on the masters 70, 70 can be arranged toprovide functions related to auxiliary information selection, placement,orientational and positional movement, for example, to draw, or “suck”,the masters to an icon when an associated cursor is within apredetermined area around the icon, and/or the like. Refer to Applicantsco-pending U.S. patent application Ser. No. 09/398,507, entitled “MasterHaving Redundant Degrees of Freedom,” filed Sep. 17, 1999, the fulldisclosure of which is incorporated herein by reference, for furtherinformation in connection with master control. Whatever method and/ordevice used to make such selection, the selection step is indicated inthe block 324 at 326 and as indicated by the dashed lines 327. It willbe appreciated that the block 324 represents selection and regulationsteps that are performed by means of the appropriate inputs, such as themaster control devices 70, 70, at the surgeon's console 12 by theoperator.

The steps whereby the information from the information sources 312 isselected and then presented or placed in the image at the video displaywill now be described in greater detail.

As mentioned, the selective placing of the auxiliary information fromthe sources 312 can be selectively caused to be displayed to extend atleast partially across an image display area of the viewer 14, such asin a localized window. When displayed on the display area, the positionat which the information is displayed relative to the display area canbe regulated or changed by the operator in two dimensions. Once adesired source is selected by the operator by operation of anappropriate input at 326, the desired source is selected at 318. Theinformation from that selected source is then forwarded to atwo-dimensional transform indicated at 328, as indicated by arrow 330.After the two-dimensional transform step at 328, the information is fedto a video mix and fade step at 332, as indicated by arrow 334. At theblock 332, the information from the selected source at 312 is mixed withthe video image captured by the endoscope 24. The video image capturedby the endoscope 24 is indicated by arrow 336. When the information fromthe selected source at 312 is thus mixed with the image captured by theendoscope 24, the combined images are forwarded to the video display asindicated by arrow 338 so that both images are placed in the image atthe viewer 14.

Referring to FIG. 12 of the drawings, an image comprising a combinationor merger of the image from the endoscope and the selected source at 312is indicated generally by reference numeral 311. An image derived fromthe source at 312 is indicated at 318, and is shown as being overlaid onthe image from the endoscope indicated at 317. A row of icons isindicated by reference numerals 315. The source at 312 was selected byactuating a corresponding one of the icons 315.

Referring again to FIG. 11 and as indicated by the dashed line 140, thesurgeon or operator of the system 10 can regulate the two-dimensionaltransform at 328, as indicated at 342. This can be achieved in anyappropriate manner, such as through appropriate input devices such as,for example, buttons, toggles, joysticks, mice, and/or the like.Advantageously, one, or both, or either, of the master control devices70, 70 are used as the input device or devices whereby thetwo-dimensional transform 328 can be regulated. The representation ofthe combined images can be presented such that the information from theselected source 312 is cropped in a localized window, as indicated inFIG. 12 of the drawings, in the image displayed at the viewer 14.Accordingly, the image 317 captured by the endoscope 14 is positioned toextend across at least a major part of the display area, the informationfrom the selected source at 312 being positioned in a localized windowoverlaid on the image captured by the endoscope 24. By manipulation ofthe input at 342, the two-dimensional transform at 328 is regulated tocause the window displaying the information from the selected source at312, to be moved relative to the rest of the image, and to be placedwhere desired by the operator, as indicated by arrows J and K in FIG.12. Typically, the size of the window can be varied, as well as itsposition relative to the rest of the image, as indicated by arrows L andM.

The video mix and fade step 332 is also regulatable by, for example, theoperator at the operator console 12, or by another person, at adifferent location, if appropriate. An appropriate input for performingsuch regulation is indicated at 344 and is operatively connected asindicated by the dashed lines 345 to the video mix and fade block at332. By manipulation of the input at 344, the information from thesource at 312 can be faded relative to the image from the endoscope 24.Advantageously, the input at 344 is also performed by means of one, orboth, or either, of the master controls 70, 70.

Referring now to the information sources at 314, these sources providetwo dimensional information which, when displayed on the display area atthe viewer 14, can be regulated so as to change the position of suchinformation relative to the display area at the viewer in threedimensions, as described in greater detail herein below.

An appropriate one of the sources of two-dimensional information at 314can be selected in similar fashion to the selection of one of thesources at 312. Accordingly, the operator can select information from adesired source at 314 by manipulating the appropriate input at 326. Theselection step is indicated at 320. Once selected, the information fromthe desired source is forwarded to a two-dimensional tothree-dimensional transform indicated at 346. At the step 346, thetwo-dimensional information from the selected source at 314 is convertedto a three-dimensional representation. It is then passed through thethree-dimensional transform indicated at 348. The three-dimensionaltransform at 348 is regulatable by the operator as indicated at 350 andby the dashed line 352. This can typically be achieved by means of anyone or more of the inputs mentioned above. However, advantageously, theappropriate input is one, or both, or either, of the master controls 70,70. By means of the input at 350, typically the position, orientationand scale of the two-dimensional information from the selected source at314, can be regulated to change its position, orientation and scale inthree dimensions. It will be appreciated that, in this fashion, not onlythe position, but also the orientation of the two-dimensional image asdisplayed in the image as viewed at the viewer 14 can be changed.

Once the operator has regulated the two-dimensional information by meansof the three-dimensional transform at 348, the information is passed toblock 354, where the information is transformed from a three-dimensionalrepresentation into a two dimensional representation. Thetwo-dimensional transform is indicated at 356. The two-dimensionaltransform is regulatable by the operator through the input 342 so as tochange the position of the information, as displayed in the image at theviewer 14, in two dimensions. It will be appreciated that thiscorresponds to changing the position of the image of the auxiliaryinformation from the source at 314 relative to the image of the surgicalsite. After regulation at 356, the information is passed to a video mixand fade block at 358, where it is mixed with the image from theendoscope 24 as indicated by arrow 336. As in the case with the videomix and fade block 332, the operator can cause the information to faderelative to the image captured by the endoscope 24 by means of the inputat 344. The image 336 from the endoscope 24 is combined with theinformation from the selected source at 314 and is then forwarded to theviewer 14 to be displayed thereon.

Referring to FIG. 13A of the drawings, an image comprising a combinationor merger of the image from the endoscope and the selected source at 314is indicated generally by reference numeral 321. An image derived fromthe source at 314 is indicated at 323 and is shown as being overlaid onthe image from the endoscope indicated at 327. As in the case withreference to FIG. 12, and as can best be seen in FIG. 13B of thedrawings, the image from the source 314 can be repositioned withreference to arrows J and K and can be adjusted in size as indicated byarrows L and M. This is achieved by the operator of the system 10 at 342by means of the transform at 356 as indicated by dashed line 140.

In addition, and with specific reference to FIG. 13B of the drawings,the image from the selected source at 314 is orientationally adjustableor regulatable. Accordingly, the image from the selected source 314 canbe regulated so as to change its orientation in three dimensions withreference to the arbitrary reference frame indicated in dashed lines inFIG. 13A. Although in FIG. 13B the image from the source 314 is shown ashaving been adjusted angularly about an arbitrary y axis with referenceto the reference frame in FIG. 13A, it will be appreciated that angularadjustment about the x and z axes can be performed in similar fashion.Such angular regulation of the image from the selected source at 314 isachieved by the operator of the system 10 at 350, so as to regulate theinformation from the selected source at 314 by means of the transform at348 as indicated by dashed line 352. In similar fashion, the image canalso be moved “inwardly” and “outwardly” as indicated by arrows Q alongthe z-axis.

Referring now to the three-dimensional information sources 316,information from one or more of the sources can be selected by theoperator by means of the input 326 and as indicated by the block 322.The three-dimensional information from the selected source at 316 isthen passed to a three-dimensional transform as indicated at 362. Theoperator, by using the input device at 350, can then regulate thisinformation in the three-dimensional transform at 362 so as to varytypically the orientation, position and scale of an image derived fromthe selected source and as displayed at the viewer 14 in similar fashionas described above with reference to FIGS. 13A and 13B. Once theinformation has been regulated in this fashion, the information isforwarded to a block 364 where the three-dimensional information istransformed from three dimensions to two dimensions. The resultanttwo-dimensional information is then forwarded to a two-dimensionaltransform at 366. The information can then again be regulated by theoperator by means of the input device at 342 as herein before describedwith reference to the two-dimensional transforms 328, 356. As before,the resultant information is then fed to a video mix and fade block asindicated at 368 where the information is mixed with the image from theendoscope and is then passed to the viewer. Where appropriate, theinformation can bc caused automatically to register with a correspondingsurgical site image captured by the endoscopc as already describedherein above. Instead, as described above, registration can be manualonly, or a combination of manual and automatic methods.

It will be appreciated that the above methods can be used withtwo-dimensional single channel video display or with three-dimensionaldual channel video display. In the latter case, the real time videosource 336 can comprise two separate images for “right” and “left”channels for viewing by the right and left eyes of the surgeon. Elements354 and 364 can then provide two separate images from two distinctviewpoints for the right and left channels respectively. The subsequentelements, or steps, can then be applied to both channels. Furthermore,element 328 can be arranged to duplicate the signal 334 into a left anda right channel and to shift them relative to each other to place theoriginal two-dimensional image in a three-dimensional viewer at variableapparent depths.

Advantageously, at least one of the master controls is operativelyarranged to fulfill some, preferably all, of the functions in the block324. Accordingly, the operator need then not remove his hands from themaster control devices 70, 70 when selecting and changing the position,orientation and scale of the auxiliary information when displayed in theimage at the viewer 14. In this way, continuity of control of thesurgical procedure is enhanced whilst still enabling the operator toaccess and place auxiliary information from one or more of the sources312, 314 and 316.

As already mentioned, the masters 70, 70 are normally operativelyassociated with the slaves. Typically, when one, or both, or either, ofthe masters are to be used selectively to place an image correspondingto auxiliary information from a selected source 312, 314, 316 in theimage or scene of the surgical site, the operative association betweenthe master, or masters, and the slaves is temporarily interrupted. Whenthis occurs, the slaves are typically held or locked in stationarypositions at the surgical site. Accordingly, the slaves are locked inthe positions they occupied immediately before disassociation with themasters 70, 70. The master or masters are then freed to enable them tobe used to select and place the desired auxiliary information in thescene or image of the surgical site captured by the endoscope 24 anddisplayed across the display area of the image display or viewer 14.Once the auxiliary information has been selected and placed, operativeassociation between the masters 70, 70 and the slaves is re-establishedto permit the operator to proceed with the surgical procedure withreference to the auxiliary information now displayed on the display areaof the viewer 14 after having been selected and placed in the scene bymeans of one, or both, or either, of the masters 70, 70. Refer toApplicant's co-pending U.S. patent application Ser. No. 09/398,960,entitled “Repositioning and Orientation of Master/Slave Relationship inMinimally Invasive Telesurgcry,” filed Sep. 17, 1999, the fulldisclosure of which is incorporated herein by reference, for a moredetailed explanation of how the operative association between themasters and the slaves is preferably reestablished.

When one of the masters is used to select the desired auxiliaryinformation, a cursor is typically generated in the image upondisassociation with the slaves. The cursor is then guided by movement ofthe master until the cursor is over the desired icon 315. The master isthen also typically used to actuate the icon to cause the desiredauxiliary information to be accessed and placed in the image of thesurgical site. Advantageously, this can be achieved by squeezing thepincher formation 86, “pressing” the icon, or the like. When placed, themaster, or both masters, is then used to vary the position and/ororientation of the image corresponding to the selected auxiliaryinformation relative to the image of the surgical site as describedabove, and where appropriate. One or both masters may be used to varythe position and orientation of auxiliary information, overlays andwindows in a manner similar to the way in which masters are used to varythe position and orientation of an image from an image capture device asdescribed in co-pending application entitled “Image Shifting Apparatusand Method for a Telerobotic System,” previously incorporated herein byreference. Of course, the present invention also encompasses othermanners of manipulating auxiliary information, in addition to thepreferred masters disclosed, such as by repositioning/rotating ajoystick, using multiple input buttons to indicate the desiredmanipulation, or using a voice control/recognition system to command thesystem to manipulate the auxiliary information as desired.

Should, during the course of a surgical procedure, an image capturedevice generating a real time video image 336 be moved, the imagedisplayed on the image display may be caused to shift and/or rotate inresponse to such image capture device movement. Instead, the video image336 can be caused to shift/rotate electronically, for example. Duringsuch a change in the displayed real time image, the two-dimensional andthree-dimensional transforms 328, 348, 354, 356, 362, 364, 366 can bearranged to synchronize their operation with the change in the displayedimage so as to cause the auxiliary information to appear attached to thedisplayed real time image. Instead, the transforms can be arranged toignore the change in the displayed real time image to cause theauxiliary information to appear attached to the image display and todrift relative to the changing real time image.

Another source of auxiliary information will now be described withreference to FIG. 14. Such a source of auxiliary information cantypically include an appropriate image gathering device such as oneincluding a transmitter and receiver arrangement, as schematicallyindicated at 413. An example of such a device is an ultrasoundtransducer which will be used by way of example only in the descriptionwhich follows. Accordingly, the invention is not to be limited to anultrasonic device. Any appropriate device which can gather similarinformation falls within the scope of the invention. Such a source canbe used to obtain a preoperative or intraoperative two-dimensional orthree-dimensional image, or model, corresponding to a surgical procedureto be performed. Accordingly, it can be either a two-dimensional source312, 314 or a three-dimensional source 316 depending on its application.As a two-dimensional source, the ultrasonic transducer can be used toobtain a single ultrasound image. As a three-dimensional source it canbe used to obtain a plurality of spaced ultrasonic images, or cuts,thereby to provide sufficient information for construction of athree-dimensional model. Accordingly, it can be arranged to move, orsweep, across a surgical site to capture such images, or cuts. This cantypically be achieved, for example, in accordance with a pre-programmedsequence for moving the ultrasound transducer, manual movement of theultrasound transducer, or the like. The ultrasonic transducer can bemounted at an end of a shaft to enable it to be introduced to thesurgical site through a relatively small aperture, in a minimallyinvasive manner. The sweeping movement can be performed manually bymoving an opposed end of the shaft positioned outside the body. To thisend, a handle can be provided on the opposed end of the shaft.Conveniently, manually operable actuators can be provided at the handleto enable the ultrasonic transducer, or probe, to be moved relative tothe end of the shaft on which it is mounted by manipulating theactuators. Instead, the shaft can be mounted on a robotic arm, themovement being controlled through a master control device. In anotherembodiment, the movement of the ultrasonic transducer can be controlledby means of a computer program. Accordingly, whether performed manuallyor automatically, a plurality of separate images can be obtained andused to form a “mosaiced” surface of images in a fashion similar to thatknown in the satellite and undersea imaging industries, namely, by“painting” the sensor, or ultrasonic transducer, across the surfacebeing viewed. Said surface of images may be intrinsically two- orthree-dimensional in nature depending on the movement of the sensorduring the build-up of the image. A different series of image “slices”may be constructed from a sensor that produces a planar image and thatis moved substantially normal to the image plane to produce a series ofslices, as is known, for example, in prenatal ultrasonic imagingpractice. Taken together, these form an intrinsically three-dimensionalor volumetric image.

These built-up two- and three-dimensional images may then be introducedinto the system to be selectively overlaid and positioned within thesurgeon's field of view at the viewer. As can best be seen in FIG. 14,such an ultrasonic image, when in a two-dimensional format, may bedisplayed as indicated by reference numeral 411.

Such a source can also be used inter- or post-operatively. For example,it can be used as a flow probe, or the like, to enable the surgeon, forexample, to ascertain the degree of fluid flow through a vessel, or thelike. In such a case, when, for example, an anastomosis procedure hasbeen performed, a surgeon, or operator, of the system may wish todetermine whether or not the anastomosed vessels are allowing sufficientblood flow therethrough, whether or not one or more of the vessels hasbeen damaged during the procedure so as to require further correctivesurgery, and/or the like. The flow probe, or ultrasonic transducer, canthen be used to establish this.

Advantageously, the ultrasonic transducer, or other appropriate device,or flow probe, can be mounted on an end of a shaft 415 to permit it tobe introduced into a patient body in similar fashion to the surgicalinstruments 28, in a minimally invasive manner. Conveniently, theultrasonic transducer can be mounted on an end of the shaft by means ofa wrist member 417 similar to the wrist member 52, or more than onewrist member, which cooperate with each other, to enable it to beangularly displaced relative to the shaft in similar fashion to the endeffector 60, in multiple degrees of freedom of movement. The mounting ofthe ultrasonic device on the end of the shaft 415, whether by means ofone or more wrist members, or otherwise, is preferably such as toprovide the ultrasonic device with relatively large sweeping movementcapability relative to the end of the shaft, as indicated by arrows 419.Accordingly, it can have a relatively large lateral range of motionalthough narrow ranges of motion, or none at all, relative to the end ofthe shaft, fall within the scope of the present invention. Movement ofthe ultrasonic device relative to at least the end of the shaft ispreferably controlled from outside the patient body, in use. For exampleactuators positioned remote from the end on which the ultrasonictransducer is mounted may be used to control movement of the ultrasonicdevice relative to the end of the shaft from outside the patient body.Instead, or in addition, actuators can be provided to cause theultrasonic transducer to scan an area of interest. The shaft may have ahandle at its proximal end, opposed from the flow probe, for manualcontrol by means of manually controllable actuators, or it may bemountable on a robotic arm as described above for control by means of amaster control device. Accordingly, in a preferred embodiment, theultrasonic device is mounted on a distal end of a robotic surgical toolof the type disclosed in U.S. Pat. No. 5,808,665, entitled “EndoscopeSurgical Instrument and Method For Use,” the full disclosure of which isincorporated herein by reference. Movement of the ultrasonic transduceracross a desired area of interest could then be accomplished by asurgeon or operator of the system 10 by manipulation of a remotelycontrolled master control at the control station 12 as described in U.S.patent application Ser. No. 09/398,507. Instead, the probe could bearranged to be releasably grasped by a surgical instrument having anappropriate complimentary end effector.

Another application of the information gathered by such an ultrasoundprobe, or the like, is to collect preoperative data on the patient, atthe surgical site, for example. Such preoperative data can then be usedto determine a location of, for example, a stenosis, or blockage, or thelike, in a blood vessel that is to be anastomosed during a heart bypassoperation for example. The auxiliary information can then be overlaid onthe “live” image of the surgical site to indicate to the surgeon wherethe surgeon should conduct the anastomosis. Conveniently, and as alreadydescribed, markers or identifiers can then be attached to the locationof the stenosis such that, should the displayed image be changed, suchas, for example by moving the endoscope, the markers or identifiersremain in a registered condition with the stenosis so that the locationof the stenosis remains clearly indicated in the displayed image.

Another source of typically two-dimensional information will now bedescribed with reference to FIG. 15 of the drawings. In FIG. 15, likereference numerals are used to designate similar parts unless otherwisestated. FIG. 15 shows a surgical instrument 28 extending through acannula 429 mounted in an aperture extending into the patient body. Atthe end of the cannula 429, schematically indicated by reference numeral431 is an image capture device. The image capture device can be in anyappropriate form. For example, it can include optical fiber bundles, anannular image capturing ring, or a suitable microcamera, or the like. Bymeans of the image capture device 431, an image extending along, thelength of the shaft 28.1 can be captured. This image can then bedisplayed selectively on the displayed image at the viewer. In thisfashion, the surgeon can be provided with images from various vantagepoints which may provide him or her, with information otherwise notreadily discernable. For example, by mounting such an image capturedevice directed along the shaft of the instrument, collisions betweeninstrument shafts and the like can be avoided.

In accordance with another aspect of the invention, auxiliaryinformation is provided to the surgeon to enable him or her to determinethe orientation of the captured image displayed on the viewer relativeto a frame of reference. For example, the auxiliary information canindicate to the surgeon which way is “up” relative to the displayedimage or scene. Such auxiliary information can be presented in anyappropriate manner, for example, by means of an icon, image,representation, or the like. One way of achieving this is by displayinga horizon “bar” or “line” in the image. Obviously, a horizon line willonly be useful to a surgeon if the endoscope is not parallelto/coincident with the gravity vector. Otherwise, rotation of theendoscopic image will simply result in rotation of the image on a planenormal to gravity. Accordingly, when the “bar” is in a “horizontal”position in the image, the top of the displayed image is either up, ordown. Preferably, the icon will include an image to enable the surgeonto distinguish between “up” and “down,” such as a dot on the “up” sideof the bar. This type of absolute registration of the horizon line,however, is not always necessary during surgery. If the image ischanged, e.g., by rotating the endoscope, the bar is caused to rotate soas to indicate that “up” in the image no longer corresponds withgravitational “up”. Therefore, the surgeon can be shown where he or sheis with respect to an artificial “horizon” bar or line overlayed on theimage captured by the endoscope.

This line or bar is preferably displayed to be perpendicular to thedirection of gravity and perpendicular to the endoscope line of sight atits viewing end. This can be achieved by taking the cross product of thegravity vector expressed in a “world frame” with the optical(longitudinal) axis of the endoscope or camera rotation matrix, which isalso expressed relative to the world frame. Therefore:

${\,^{world}H} = {{\,^{world}\begin{bmatrix}0 \\0 \\{- 1}\end{bmatrix}}X^{world}z_{endoscope}}$

where

-   -   H(world)=horizon vector in the world reference frame, and    -   z(endoscope)=optical axis vector of endoscope.

This horizon vector, H(world), is preferably represented in the camerareference frame, T(endoscope)—the reference frame attached to the distaltip of the endoscope whose z—axis is its optical axis—by transposing thecamera frame as follows:^(endoscope) H=(^(world) T _(endoscope))^(T)·^(world) H

where H(endoscope) is the horizon vector in the endoscope referenceframe. In order to facilitate displaying the horizon line at any desiredlocation on the image display screen, rather than with its origin at aparticular image location such as the origin of the x and y-axes of thecamera/endoscope frame, only the angle relative to one of the x- andy-axes of the camera/endoscope frame—preferably the x-axis—need to betracked. This angle can be determined by the geometric relationshipbetween the x and y portions of H(endoscope) vector. Thus:

${Angle} = {{arc}\mspace{14mu}{\tan( \frac{{\,^{endoscope}H}(y)}{{\,^{endoscope}H}(x)} )}}$

If it is desirable to distinguish between “up” and “down” in the image,calculation of the angle will also need to take into account whethereither or both of the H(x) and H(y) values are positive or negative.

The artificial horizon or “bar” can then be displayed in the displayedimage as indicated at 510 in FIG. 16A. If the endoscope is then rotatedto rotate the displayed image in a clockwise direction as indicated byarrow B1, the line or bar 510 follows as indicated in FIG. 16B. Insteadof a line or bar, an arrow corresponding with the direction of gravitycan be used, in the manner of a virtual “plumb line” for example.Instead, and with reference to FIG. 17A, opposed indicators or cursors520, 522 can be used. As the image is rotated clockwise, as indicated byarrow B2, for example, the markers are moved along with rotation of theimage as indicated in FIG. 17B. The surgeon can then determineorientation with reference to an imaginary line extending between themarkers 520, 522.

Another form of auxiliary information that can be displayed on thedisplayed image is one or more force indicators to indicate to theoperator the degree of force applied between the fingers of the endeffectors 60, for example. This can be achieved by measuring associatedmotor current for example, and can be in the form of a graph (such as abar graph) associated with each of the end effectors and displayed onthe image for the surgeon to see. The graph may include a variety ofcolors, with lighter forces indicated by both different colors as wellas changes in magnitude of the graphic display. The graphs may indicateforce direction as well as magnitude for example. A zeroing function canbe included to negate effects of other forces on the end effectors suchas if they contact tissue, or body walls, or the like.

While the above is a complete description of preferred embodiments ofthe invention, various alternatives, modifications and equivalents maybe used. It should be evident that the present invention is equallyapplicable by making appropriate modifications to the embodimentsdescribed above. Therefore, the above description should not be taken aslimiting the scope of the invention which is defined by the metes andbounds of the appended claims along with their full scope ofequivalents.

1. A robotically controlled surgical system comprising: a surgicalinstrument; an image capture device positionable so as to capture animage of a surgical site; at least one image display coupled to theimage capture device so as to display the image of the surgical sire; amaster control input device; a source of auxiliary information; and atleast one processor coupled to the surgical instrument, the imagecapture device, the at least one image display, the master control inputdevice, and the source of auxiliary information, and coupling the mastercontrol input device to the surgical instrument and the source ofauxiliary information so as to allow manipulation of the surgicalinstrument in response to manipulation of the master control inputdevice while the master control input device is operatively associatedwith the surgical instrument, and allow manipulation of the auxiliaryinformation on the image display in response to manipulation of themaster control input device while the operative association between themaster control input device and the surgical instrument is interrupted.2. The system of claim 1, wherein the source of the auxiliaryinformation is a data capturing/gathering device.
 3. The system of claim2, wherein the auxiliary information is captured in real-time.
 4. Thesystem of claim 1, wherein the source of the auxiliary information is adata storage device.
 5. The system of claim 4, wherein the auxiliaryinformation comprises preoperative information specific to a surgicalprocedure to be performed.
 6. The system of claim 4, wherein theauxiliary information comprises preoperative information specific to thepatient on which a surgical procedure is to be performed.
 7. The systemof claim 1, wherein the processor is configured to adjust an image ofthe auxiliary information in a discrete window on the image display. 8.The system of claim 7, wherein the adjusting of the auxiliaryinformation in a discrete window on the image display further comprisesvarying the position of the discrete window across the image display. 9.The system of claim 7, wherein the adjusting of the auxiliaryinformation in a discrete window on the image display further comprisesvarying the dimensions of the discrete window across the image display.10. The system of claim 7, wherein the adjusting of the auxiliaryinformation in a discrete window on the image display further comprisesvarying the orientation of the discrete window across the image display.11. The system of claim 1, wherein the manipulation of the auxiliaryinformation on the image display comprises displaying the auxiliaryinformation on the image display such that the auxiliary information isoverlaid on the image of the surgical site.
 12. The system of claim 1,wherein the auxiliary information comprises an image corresponding tothe surgical site, and wherein the manipulation of the auxiliaryinformation on the image display involves alternately displaying theauxiliary information image and the image of the surgical site on theimage display.
 13. The system of claim 1, wherein the auxiliaryinformation comprises an image corresponding to the surgical site, andwherein the manipulation of the auxiliary information on the imagedisplay comprises at least generally merging the auxiliary informationimage with the image of the surgical site so that the images aresuperimposed and are both visible on the image display.
 14. The systemof claim 1, wherein the auxiliary information comprises a memberselected from the group consisting of a CAT image, an MRI image, anX-ray image, and an ultrasound image.
 15. The system of claim 1, whereinthe auxiliary information comprises a member selected from the groupconsisting of an ECG signal, an EEG signal, a blood pressure signal, anda heartbeat signal.
 16. The system of claim 1, wherein the auxiliaryinformation comprises a member selected from the group consisting of awarning message, a clock indicating time, an e-mail service, atemperature reading, oxygen levels in a patient's blood, and informationfrom a remote computer.
 17. The system of claim 1, wherein the auxiliaryinformation comprises a member selected from the group consisting of astreaming video corresponding to a surgical procedure to be performed, ageneric anatomical image, and a patient specific anatomical image. 18.The system of claim 1, wherein the auxiliary information comprises amember selected from the group consisting of a three-dimensional model,a raw volumetric image, a computer generated model, and a computergenerated image.
 19. The system of claim 1, wherein the auxiliaryinformation comprises a member selected from the group consisting of animage of the surgical instrument, an image of the surgical instrumentlocation, and an image of art entry port location through which thesurgical instrument extends.
 20. The system of claim 1, wherein theauxiliary information is an indication of the orientation of the imageof the surgical site relative to a frame of reference.
 21. The system ofclaim 1, wherein the image capture device captures a real time image ofthe surgical site, and wherein the auxiliary information comprises animage of the surgical site which is more panoramic than the real timeimage of the surgical site such that the auxiliary information image asmanipulated on the image display extends around a periphery of thereal-time image of the surgical site.
 22. The system as claimed in claim1, wherein the auxiliary information comprises an image corresponding toa constraint, and wherein the auxiliary information when adjusted on Theimage display is superimposed with the image of the surgical site suchthat the image of the constraint is arranged to indicate that movementof the surgical instrument through the image of the constraint is eitherto be avoided or not possible.
 23. The system as claimed in claim 22,the processor adjusting the image of the constraint relative to theimage of the surgical site in response to the manipulation of the mastercontrol input device.
 24. The system as claimed in claim 23, wherein themaster control input device can be selectively operatively associatedwith the image of the constraint, said processor adjusting the image ofthe constraint relative to the image of the surgical site comprisingplacing the image of the constraint relative to the image of thesurgical site in a location corresponding to a position beyond which thesurgeon desires not to move any surgical tool.
 25. The system as claimedin claim 1, wherein the image capture device captures a real time imageof the site, the system further comprising a second image capture devicethat captures a second image related to the surgical procedure from avantage point different to a vantage point from which the real timeimage of the site is captured, the processor adjusting the second imageon the image display in response to the manipulation of the mastercontrol input device.
 26. The system of claim 1, wherein themanipulation of the auxiliary information on the image display comprisesa least one of positionally adjusting the auxiliary information,orientationally adjusting the auxiliary information, or scaling theauxiliary information.
 27. The system of claim 1, wherein the processoris configured to effect master/slave movement of the surgical instrumentin response to a first manipulation of the master control input device,and to adjust the image of the auxiliary information on the imagedisplay in response to a second manipulation of the master control inputdevice.
 28. A method of preparing for or performing a robotic surgicalprocedure on a patient, the method comprising: a) displaying an image ofauxiliary information relevant to the surgical procedure on an imagedisplay; b) effecting a master/slave movement of a surgical instrumentof the robotic surgical system using a master control input device whilethe master control input device is operatively associated with thesurgical instrument; and c) adjusting the image of the auxiliaryinformation on the image display of the robotic surgical system usingthe master control input device while operative association between themaster control input device and the surgical instrument is interrupted.29. A method of preparing for or performing a robotic surgical procedureon a patient, the method comprising: a) viewing an image of a surgicalsite on an image display; b) effecting a master/slave movement of asurgical instrument of the robotic surgical system using a mastercontrol input device while the master control input device isoperatively associated with the surgical instrument; and c) selectivelyaccessing a source of auxiliary information, displaying the selectedauxiliary information on the image display, and manipulating theauxiliary information on the image display using the master controlinput device while operative association between the master controlinput device and the surgical instrument is interrupted.